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ABSTRACT 

The purpose of this project was to construct a valid 
and reliable noncurriculum specific measure of integrated science 
process skills intended for use with middle school students. The 
major efforts in test development were focused on the refinements and 
modifications of the set of objectives and test items assessed by the 
existing Middle Grades Integrated science Process Skills Test (Cronin 
and Padilla). Skills associated with planning, conducting, and 
interpreting results from investigations were delineated, and six 
objectives encompassing these skills were identified. There were 21 
items judged to be the best measures of these objectives chosen for 
inclusion in the test. The POPS test was constructed in order to 
assess the process skills dealing with identifying experimental 
questions, identifying variables, formulating hypotheses, designing 
investigations, graphing data, and interpreting data. The readability 
of the test was assessed and an average grade level index of 6.8 was 
obtained. The POPS test is a reliable instrument for diagnostic 
and/or summative assessmant in science classes or research studies. 
The test may be a useful means for classroom-based research, 
evaluation of instruction and learning, and curriculum validation in 
evaluation as well as assessing process skills competence of middle 
school students. (MVL) 
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DEVELOPMENT OF THE PERFORMANCE OF PROCESS SKILLS (POPS) TEST 
FOR MIDDLE GRADES STUDENTS 



Abstract 



The purpose of this project was to construct a valid and reliable 
non-curriculum-specific measure of integrated science process skills intended 
for use with middle school students. 

The major efforts in test development were focused on the refinements and 
modifications of the set of objectives and test items assessed by the 
existing Middle Grades Integra ted Science Process Skills Test (Cronin and 
Padilla, 1986). Each objective and item was carefully analyzed to ensure 
that it was a valid representation of the intended process skills. Skills 
associated with planning, conducting, and interpreting results from 
investigations were delineated, and six objectives encompassing these skills 
were identified. Twenty-one items judged to be the best measures of these 
objectives were chosen for inclusion in the test. Modifications were made on 
the items in order to provide additional explanation for specific terms, 
i.e., the manipulated, responding, and controlled variables. The POPS test 
was constructed in order to assess the process skills dealing with 
identifying experimental questions (three items), identifying variables (six 
items), formulating hypotheses (three items), designing investigations (three 
items), graphing data (three items), and interpreting data (three items). 
The readability of the test was assessed using the FOG index, and the average 
grade level index of 6.8 was obtained. To establish content validity, 
objectives and items were submitted to a panel of reviewers. The responses 
of the four experts were consistent on almost all items in terms o* 
indicating the correct answer and keying to a process skills objective. This 
concurrence of reviewers was taken as evidence of content validity and 
objectivity of scoring. 

To examine test reliability and item indices, tie POPS test was 
administered to middle school students in grades six through eight. A total 
of 1,402 students were involved in this field test. Total scores on the 
21-item test for overall students ranged from 1 to 20 (Mean = 9.8, S.D. = 
A. 2). Total test reliability (KR-20) was .75. Item difficulty indices 
ranged from .28 to .79 with an average of .47. Item discrimination indices 
obtained by using the upper 27Z and lower 27Z of the sample group showed that 
20 of the 21 items were above .30 with an average of .49. Point biserial 
correlation indices of item discrimination showed that 19 of the 21 items 
were above .30 with an average of .41. Each of these inUices fell well 
within the acceptable range for a reliable test. 

These results provide evidence of the reliability and validity of the 
test. The POPS test is a reliable instrument for diagnostic and/or summative 
assessment in science classes or research studies. The test may be a useful 
means in classroom-based research, evaluation of instruction and learning, 
and curriculum validation in evaluation as well as assessing process skills 
competence of middle school students. 
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DEVELOPMENT OF THE PERFORMANCE OF PROCESS SKILLS (POPS) TEST 



FOR MIDDLE GRADES STUDENTS 



Purpose of This Study 

The primary purpose of this project was to construct a valid and reliable 
non-curriculum-specific measure of integrated science process skills intended 
for use with middle school students. 

At the middle school level, integrated science process skills such as 
identifying experimental questions, indent ifying variables, formulating 
hypotheses, designing investigations, graphing and interpreting data are a 
vital aspect of meaningful science laboratory activity. These process skills 
represent the rational and logical thinking skills required in the process of 
problem solving in science. Few paper and pencil instruments to assess the 
process skills performance for middle grades students exist; thus, the 
present development took place. 

The new test could serve either as an alternate and equivalent process 
skills test or enlarge the pool o ; available items for process skills 
assessment in middle school grades. This test could be used for diagnostic 
or summative assessment in science classes or research studies in ways such 
as the following: 

1. As sessing the process skill competency level possessed by students 

2. Assessing the student acquisition and progress measured by process 
skill performance 

3. Evaluation of the effectiveness of certain instructional strategies, 
teaching skills, curriculum materials or modules on the development 
of process skills in students 

4. An alternative to a practical laboratory procedure as a way to 
assess process skill performance 



Background and Need for Study 

Development of science process skills in students has been considered to 
be one of the major goals of science education (e.g., Gagne, 1965; Hernon, 
1970; Neie, 1972; Okey, 1972; Padilla, 1980). In the past quarter century, a 
significant feature of most of the science curriculum development was a major 
shift of emphasis away from the teaching of science as a body of established 
knowledge toward science as a human activity, with increasing emphasis on 
experience of the processes an*i procedures of science (Hodson, 1988). The 
acceptance of process learning has created a need for measures of process 
skill performance. 

Since the 1960's, as lagging behind the curriculum development which 
emphasizes the process of science, a number of paper and pencil measures have 

* 
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been developed to assess science process skills performance. Initially, 
these efforts wpre aimed at the assessment of process skills associated with 
specific cur-icula such as the Science--A Process Approach (SAPA), the 
Science Curriculum I mprovement Stu dy (SCIS), and the Biological Science 
Curriculum Stuuv (BSCS) (BSCS, 1962; Walbesser et. al., 1965; Fyffe, 1972; 
Burns, 1972; Riley, 1972; McLeod et. al., 1975; Ludeman, 1975). 

In an attempt to separate a process skills test from a specific 
curriculum, several efforts have been done. Some have focused on measuring 
all science process skills including both integrated and basic process skills 
(e.g., Tannenbaum, 1968). Others were developed to measure a set of the 
basic process skills for upper elementary and lower secondary school students 
(Molitor and George, 1976; Padilla, Cronin, and Twiest, 1985). Recently, 
several attempts have been made to measure the integrated science process 
skills (Dillashaw and Okey, 1980; Tobin and Capie, 1982; Burns, Wise, and 
Okey, 1983; McKenzie and Padilla, in press; Cronin and Padilla, 1986). Most 
of these existing integrated process skills tests are geared primarily for 
upper secondary school students and contain items which require specific 
knowledge and/ or well developed reading skills. Few tests of integrated 
process skills can be found suitable for middle school students; e.g., Cronin 
and Padilla (1986) developed the Middle Grades Integrated Process Skills Test 
(MIPT) intended for use with the grades 6-9 students. 

In view of the continued need and relative scarcity of integrated process 
skills tests for middle school students, the present development of the 
Performance of Process Skills (POPS) Test has taken place. The following 
criteria were outlined for the test: 

1. A measure of the integrated science process skills associated with 
the serial process of science experimentations such as identifying 
experimental questions, identifying variables, formulating 
hypotheses, designing investigations, graphing and interpreting 
data, and indeed assessing the thinking skills besides specific 
knowledge in the procedures 

2. A non-curriculum-specific measure ana content-tree test items which 
are referenced to a general content domain and do not require 
specific content knowledge in a particular context 

3. Multiple opportunities provided to demonstrate competence on each 
process skill and a wide range of difficulty of items addressing 
each process skill 

4. Appropriate amount of reading and reading difficulty for middle 
school students (6-8) and avoiding the difficulty in understanding 
the meaning of specific terms such as manipulated, responding, and 
controlled variables 

5. A multiple choice with four option response format 

6. Appropriate number of test items and test length suitable for group 
administration less than 25 minutes; e.g., the test can be 
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The readability of the POPS test was assessed using the FOG index 
(Gunning, 1975). This index predicts the reading grade level necessary to 
read with 90 to 100% comprehension. On this scale, an average grade level 
index of 6.8 was obtained. 



Field Test and Results 

For the purposes to examine test reliability and item indices, the 
modified version of the POPS test was administered to middle school students 
in grades six through eight. A total of 1,402 students from a relatively 
rural school district in an eastern part of North Carolina were involved in 
this field test. The sample group involved was heterogeneous to provide 
representation across ability levels, socio-economic levels, gender, and 
race. 




The descriptive statistics for the POPS test are shown in Table 1. The 
mean score and standard deviation for overall students on the 21-item test 
were 9.77 and 4.16 respectively. Total scores ranged from i to 20 with a 
standard error of measurement of 2.08. Total test reliability, as computed 
by the Kuder -Richardson formula (KR-20) was equal to .75. Item difficulty 
indices ranged from .28 to .79 with an average of .47. Item discrimination 
indices obtained by using the upper 27% and lower 27% of the sample group 
ranged from .27 to .71 with an average of .49 and were above .30 for 20 of 
the 21 items. Point biserial correlation indices of item discrimination 
ranged from .28 to .55 with an average of .41 and were above .30 for 19 of 
the 21 items. Each of these three test characteristics (reliability, item 
difficulty index, and item discrimination index) were well within the 
acceptable range for reliable tests (Payne, 1974). Based on these results 
from the field test, no further test revisions were considered necessary. 
Figure 2 summarizes descriptive characteristics of the POPS test. 

For additional analyses, the entire POPS test was broken down into its 
six subscales of process skill objectives. The mean scores and standard 
deviations on the POPS total and each subscalr by grade level and overall 
students are shown in Table 2. For overall students, correct response 
^ rC ^ a88S W8re hi 8 hest for the process skills of formulating hypotheses 
(59.0%) and were lowest for the process skills of graohing data (36.3%) and 
interpreting data (36.3%). These results indicated tne difficulty levels of 
each process skill objective as a subscale of the POPS. 
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Conclusions and Educational Significance 

The descriptive statistics obtained by field tests indicated tha* mean 
performance on the POPS test was comparatively low (9.8 out of 21). The 
standard' deviation of 4.2 and a range of 20 indicated that the test 
differentiated students of differing ability. This observation was supported 
by the indices of reliability, item difficulty, and item discrimination which 
provided measures of the suitability of the test scores for differentiating 
student performance. Since each of these indices fell well within the 
acceptable range for reliable tests, the POPS test can be used by teachers to 



obtain dependable student data as a basis for making decisions about 
individual students on item performance* In a research context, the test can 
be used to obtain measures to answer questions that require individual 
student performance to be differentiated or mean performance on the POPS 
items to be compared* The POPS measures are dependable for answering a range 
of research and evalution questions where performance of students , 
performance on items, or performance on different occasions need to be 
differentiated* The POPS test has applications for research and evaluation 
where teaching and learning integrated science process skills are a concert, 
if it is a valid measure of this ability* 

Support was obtained for the content validity of the POPS test* Four 
science educators as reviewers agreed that the items measured the objectives 
they were designed to measure and that the scoring procedures proposed by the 
test designers were correct. This concurrence of raters was taken as 
evidence of content validity and objectivity of scoring. In addition to 
content validity, before the POPS test could be considered a highly valid 
measure of integrated science process skill ability, more validity data would 
be necessary, including factor analysis for construct validity and 
correlation analysis with interview tasks for criterion validity. 

The data that has been gathered in this investigation provides evidence 
of the reliability and content validity of the POPS test. The search of 
available integrated science process skills tests showed the continued need 
and relative scarcity of a test geared to middle school students and not 
associated with any particular science curriculum. The development of the 
Performance of Process Skills (POPS) Test fills this void. This test could 
serve either as an alternate and equivalent process skills test or enlarge 
the pool of available items for process skills assessment in middle school 
grades. The 21 -item paper and pencil test can be administered to groups of 
students to obtain measures of process skill acquisition for use by classroom 
teachers, researchers, and evaluators. The test is not specific to a given 
curriculum or content area; therefore, it may be used across the various 
disciplines of science. As well as assessing process skills competence of 
middle school students, the test may also be a useful means in classroom- 
based research, evaluation of instruction and learning, and curriculum 
validation in evaluation. The POPS test is a reliable instrument for 
diagnostic and/or suramative assessment in science classes or research 
studies. 



* Copies of the POPS test can be obtained from either author. 
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Table 1 

Descriptive Statistics for the POPS Test (21 Items) 



Number Mean Standaid Deviation Reliability 



6 252 8.72 3.65 . 68 

7 83 * 9.23 3.98 .73 

t 

8 316 11-63 4.36 .78 
OVERALL U02 9.77 4 16 7c 



Table 2 

Means and Standard Deviations for the POPS Test 
by Process Skill Ob jctive and Grad< 

Mean 
(S.D.^ 



Grade 

6th 7th 8th Total 

Process Skill Objective (N-252) (N«834) (N-316) (N«1A02) 



PI. 


Identifying Experimental 
Questions (3) 


1.51 
(0.96) 


1.49 
(0.96) 


1.80 
(0.94) 


1.58 
(0.96) 


P2. 


Identifying Variables (6) 


2.33 
(1.45) 


2.52 
U.S1) 


3.32 
(1.68) 


2.67 
(1.58) 


P3. 


Formulating Hypotheses (3) 


1.65 
(0.97) 


1.69 
(0.98) 


2.02 
(0.90) 


1.77 
(0.97) 


PA. 


Designing Investigations (3) 


1.35 
(1.02) 


1.53 
(1.04) 


1.78 
(0.99) 


1.57 
(1.04) 


P5. 


Graphing Data (3) 


1.00 
(0 »') 


1.01 
(0.86) 


1.31 
(1.00) 


1.09 
(0.90) 


P6. 


Interpreting Data (3) 


0.89 
(0.81) 


1.02 
(0.90) 


1.42 
(0.90) 


1.C9 
(0.91) 



Overall Skills (21) 



8.72 9.23 11.63 9.77 
(3.65) (3.98) (4.36) (4.16) 



Process Skill Objectives 



Number of 
Test. Items 



PI. Identifying Experimental Questions: 

Identify a reasonable question which can be 
tested in an experiment. 

P2. Identifying Variables: 

given a description of an experiment, identify 
the manipulated, responding, and controlled 
variables involved in it. 

P3. Formulating Hypotheses: 

given a description of variables involved in 
an experiment, select a reasonable hypothesis 
which is being or can be tested. 

P4. Designing Investigations: 

given a hypothesis, select a suitable design 
for an experiment to test it. 

P5. Graphing Data: 

given a description of an experiment and 
obtained data, identify a graph that 
represents the data correctly. 

P6. Interpreting Data: 

given a description of an experim^c and 
graphed results, describe the relationship 
shown on the graph or identify * 3-raph that 
shows the relationship described. 



Figure 1 

The POPS Objectives and Number of Test Items 
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Process Skill Tested: 



Target Group: 
Number of Items: 
Response Format: 
Test Readability (FOG): 
Time to Complete Test: 
Discrimination Index: 

Difficulty Index: 

Reliability (KR-20): 
Total Test Score: 

Standard Error of Measurement: 



Identifying Experimental .Questions 
Identifying Variables 
Formulating Hypotheses 
Designing Investigations 
Graphing Data 
Interpreting Data 

Middle Grades 6-8 

21 

Four Alternative Multiple Choice 
6.8 

25 minutes 

.49 (Average) 

.27 to .71 (Range) 

20 of the 21 items above .30 

.47 Average) 

.26 to .79 (Range) 

.75 

9.77 (Mean) 
4.16 (S.D.) 
1 to 20 (Range) 

2.08 



Figure 2 

Summary Characteristics of the POPS Test 



9 

ERLC 



15 



